[DRAFT] Analysis of Software and Data Carpentry’s Pre- and Post-Workshop Surveys

Authors: The Carpentries Assessment Team

Published: July 2018

Overview

Since XXXX, Software and Data Carpentry have collected information on learner demographics, perception of tools and confidence in working with data. As we continue in our goal to streamline processes as The Carpentries, the Assessment Team completed an analysis of the pre- and post-workshop surveys for both Software and Data Carpentry. The goal of this analysis is to understand the impact our workshops are having on learners, and how we can improve our surveys and assessment infrastructure. This report covers the workshops from NA to NA.

As an overview, 1495 learners have responded to Data Carpentry’s surveys, while 14185 have responded to Software Carpentry’s.

This report includes the following:

  • Motivation for Attending Carpentries Workshops
  • Workshop Type and Perception of Workshop Environment/Experience
  • Effect of Workshops on Learner’s Self-Reported Perspectives, Skills, and Confidence
  • Ability to Perform Computing Tasks
  • Demographics
  • Summary

Motivation for Attending Carpentries Workshops

Learners attend Carpentries workshops for many reasons. Data Carpentry’s workshops are domain specific and focus on the fundamental data skills needed to conduct research. Data Carpentry’s Ecology and Social Sciences curricula begin with a lesson on data organization and includes data cleaning with OpenRefine. From there, learners spend time learning a base programming language, either Python or R.

Data Carpentry’s Genomics curriculum also includes programming, however the focus of this curriculum is best practices for organization of bioinformatics projects and data, use of command line utilities and tools to analyze sequence quality and perform variant calling, and connecting to and using cloud computing.

Why learners attend Data Carpentry workshops n %
To learn skills that I can apply to my work in the future 960 63.7
To learn skills that will help me get a job 445 29.5
As a requirement for my program/current position 103 6.8

XX% of Data Carpentry learners attend workshops to learn skills they can apply to their work in the future.

Software Carpentry workshops teach automation with the Unix shell, a tool that allows users to run commands interactively or by scripting. In Software Carpentry workshops, version control of source code with Git and GitHub are also taught for learners to learn facilitating contribution and collaboration on online repositories. Programming in R or Python is also taught. Software Carpentry’s curriculum teaches basic lab skills for scientific computing. Compared to Data Carpentry’s learners, Software Carpentry’s tend to have more experience witht the tools covered in the workshops, and learners come to learn new and/or additional topics (XX%).

Why learners attend Software Carpentry Workshops n %
To cover new/additional topics 458 74.7
To network 78 12.7
To become a Software Carpentry helper/instructor 39 6.4
To help host/run a workshop 38 6.2

The majority (XX%) of Data Carpentry’s respondents are either unsatisfied or feel neutral about being satisfied with their current data management practices. By data management practices, we include behaviors such as keeping your raw data raw, reusing code, and using databases, queries, and scripts to manage large datasets.

Data Carpentry Learners satisfaction with current data management practices n %
Very unsatisfied 100 6.7
Unsatisfied 408 27.3
Neutral 405 27.1
Satisfied 173 11.6
Very satisfied 22 1.5
Not sure 75 5.0
Not applicable 60 4.0
Didn’t answer 252 16.9

In terms of current programming usage, XX% of learners either never use programming, or use programming less than once per year, but no more than several times per year. Only XX% program on a daily basis. This is no surprise, as Data Carpentry workshops tend to attract novices.

Data Carpentry Learners current programming usage n %
Never 314 31.9
Less than once per year 151 15.4
Several times per year 168 17.1
Monthly 101 10.3
Weekly 131 13.3
Daily 118 12.0

Data Carpentry workshops are mainly populated by “word-of-mouth”. XX% of respondents learned about the workshop from either a friend, colleague, or their advisor or supervisor.

How respondents find out about Data Carpentry workshops n %
My friend/colleague told me about it 324 47.0
My advisor/supervisor told me about it 244 35.4
Read about it in a newsletter or university web site 85 12.3
Other web site 20 2.9
Twitter or other social media 17 2.5

On the contrary, the majority of Software Carpentry workshop respondents (XX%) find out about workshops through instituion mailing lists or flyers.

How respondents find out about Software Carpentry workshops n %
Institution mailing list or flyer 5152 75.3
Conference/meeting/seminar 641 9.4
Funding organization or program officer 357 5.2
Our website 357 5.2
Social Media (Twitter, Facebook, etc.) 294 4.3
Journal or publication 37 0.5

In summary, both Data and Software Carpentry workshop respondents attend workshops to learn about or improve upon their current data management and analysis skills.

Workshop Type and Perception of Workshop Environment/Experience

As previously mentioned, Data Carpentry workshops are domain specific, and curricula include Ecology, Genomics, Geospatial, Social Sciences, and Reproducible Research. XX% of respondents learned R in their workshop, while XX% learned Python.

Data Carpentry: Language Covered in Workshops n %
R 808 64.7
Python 188 15.1
I don’t know/I don’t remember 218 17.5
Neither 35 2.8

Workshop Environment

The Carpentries is committed to making participation in our workshops a harassment-free experience for everyone, regardless of who you are, where you come from, or your experience with the tools we teach. We establish norms for interaction by having, discussing, and enforcing a Code of Conduct such that our workshops provide open and inclusive learning environments. XX% of Data Carpentry respondents either agree or strongly agree that they felt comfortable learning in their workshop environment, and XX% of Software Carpentry’s respondents agreed or strongly agreed the workshop atmosphere was welcoming.

Data Carpentry respondents were asked to rate their level of agreement with several statements regarding their instructor’s knowledge, instructional method, and enthusiasm. Their responses are in the figure below, and axis labels correspond to the statements as follows:

  • Instructors Knowledge: The instructors were knowledgeable about the material being taught.
  • Instructors Interacting: I felt comfortable interacting with the instructors.
  • Instructors Enthusiastic: The instructors were enthusiastic about the workshop.
  • Instructors Clear Answers: I was able to get clear answers to my questions from the instructors.

The largest impact we see is that XX% of respondents said they felt comfortable interacting with the instructors. We know that our instructors are the reason why our workshops are so well received. It’s also great to see that XX% and XX% of respondents felt our instructors were knowledgeable about the material being taught, and were enthusiastic about the workshop, respectively. We would like to explore what training would help our instructors so that the percentage of respondents who felt they were able to get clear answers to their questions from the instructors would increase.

Software Carpentry respondents were asked to rate how they felt instructors and helpers worked as a team based on the following criteria:

  • Considerate: Instructors/Helpers were considerate.
  • Enthusiastic: Instructors/Helpers were enthusiastic.
  • Clear Answers: Instructors/Helpers gave clear answers to your questions.
  • Communicators: Instructors/Helpers were good communicators.

The two Likert plots below provide an analysis of respondent’s answers for both instructors and helpers.

From the figures above, we see that Software Carpentry instructors and helpers are considerate, enthusiastic, give clear answers to questions, and are good communicators. As a whole, our instructors work as a team and are successful in creating a warm and welcoming workshop environment.

One of the goals for Data Carpentry’s lessons is that learners are able to immediately apply what they learned at the workshop. The figure below shows that XX% either agree or strongly agree that they’re able to immediately apply what they learned.

As the majority of Software Carpentry learners attend workshops to learn new skills it is great to see that XX% of learners either learned mostly or all new information during the workshop.

Workshop Experience

We want to be proactive in ensuring learners have access to whatever they need to participate in a workshop. Both Data and Software Carpentry learners are asked to inform workshop organizers if there is anything they need that would make their workshop experience better. Data Carpentry’s respondents were asked if they had accessibility issues, and XX% reported they did. After reading the open-ended responses, we see that the issues were related to not being able to hear and/or see in the back of the room.

Data Carpentry Respondents Having Accessibility Issues n %
No 654 89.2
Yes 79 10.8

We use the Net Promoter Score to measure learners’ likelihood of recommending workshops to a friend or colleague. The scoring for this question is on a 0 to 100 scale. Respondents scoring from 0 to 64 are labeled Detractors, and are believed to be less likely to recommend a workshop. Those who respond with a score of 85 to 100 are called Promoters, and are considered likely to recommend a workshop. Respondents between 65 and 84 are labeled Passives, and their behavior falls in the middle of Promoters and Detractors.

Data Carpentry Promoter Score n %
Detractor 32 4.395604
Passive 131 17.994505
Promoter 565 77.609890

78% of Data Carpentry respondants are promoters (i.e. would recommend a workshop).

Software Carpentry Promoter Score n %
Detractor 32 4.395604
Passive 131 17.994505
Promoter 565 77.609890

For Software Carpentry respondents, NA% are promoters.

In summary, Data and Software Carpentry workshops provide a warm and welcoming environment whether learners are brand new to programming or have some experience. Attendees are recommending workshops to their friends and colleagues, and we know that our instructors and helpers are the major reason why.

Effect of Workshops on Learners Self-Reported Perspectives: Skills & Confidence

Learners were asked to rate their level of agreement with the following statements related to Data Carpentry’s workshop goals and learning objectives. The figure below provides a visual representation of their responses, comparing them before the workshop and after the workshop. Axis labels and the corresponding question are as follows:

  • Write Program: I can write a small program/script/macro to solve a problem in my own work.
  • Search Online: I know how to search for answers to my technical questions online.
  • Raw Data: Having access to the original, raw data is important to be able to repeat an analysis.
  • Programming Efficient: Using a programming language (like R or Python) can make me more efficient at working with data.
  • Programming Confident: I am confident in my ability to make use of programming software to work with data.
  • Overcome Problem: While working on a programming project, if I get stuck, I can find ways of overcoming the problem.
  • Analyses Easier: Using a programming language (like R or Python) can make my analyses easier to reproduce.

As the scoring for the above factors is ordinal from strongly disagree (1) to strongly agree (5), we show the mode (most frequent responses) for respondents’ before the workshop, and after the workshop. The comparison above is paired, meaning, we are comparing those who provided us with a unique identifier. This figure includes XX respondents.

In the figure below we show another representation of the pre- and post-comparison of respondents skills and perspectives.

As the scoring for the above factors is ordinal from strongly disagree (1) to strongly agree (5), we show the mode (most frequent responses) for respondents’ before the workshop, and after the workshop. The comparison above is paired, meaning, we are comparing those who provided us with a unique identifier. This figure includes XX respondents.

In the figure below we show another representation of the pre- and post-comparison of respondents skills and perspectives.

Software Carpentry Respondets were asked to tell us about their experience with these topics before the workshop:

  • R
  • Unix Shell
  • SQL
  • Python
  • Version Control with Git

The following is a comparison of Software Carpentry Respondents’ knowledge about the tools before compared to after the workshop.

Respondent Ability to Perform Computing Tasks

Motivation is important, but being confident in your ability to complete specific computing tasks is an equally important goal of Software Carpentry. The grid below shows respondents’ self-reported ability to complete tasks including:

  • Using pipes to connect shell commands
  • Writing a ‘for loop’ to automate tasks
  • Initializing a repository with git
  • Writing a function
  • Importing a library or package in R or Python
  • Writing a unit test in Python or R
  • Writing an SQL query

It also provides their self-reported level of confidence in being able to complete the tasks above after completing the workshop.

Demographics

In Software Carpentry’s pre-workshop survey, respondents are asked whether or not their workshop takes place in the United States.

Software Carpentry Workshops in US n %
Yes 6489 45.7
No 4650 32.8
Didn’t answer 3046 21.5
Data Carpentry’s Respondents by Discipline n %
Life Sciences 444 22.0
Agricultural or Environmental Sciences 307 15.2
Bioinformatics/Genomics 292 14.5
Biomedical/Health Sciences 288 14.3
Social Sciences 122 6.0
Mathematics or Statistics 101 5.0
Earth Sciences 96 4.8
Engineering 91 4.5
Computer Science 88 4.4
Business/Economics 57 2.8
Humanities 53 2.6
Physical Sciences 53 2.6
Library Sciences 28 1.4
Software Carpentry’s Respondents by Discipline n %
Life Science - Organismal/systems (ecology, botany, zoology, microbiology, neuroscience) 2698 21.0
Life Sciences (Genetics, genomics, bioinformatics ) 2683 20.9
Mathematics/statistics 945 7.4
Physics 803 6.2
Planetary sciences (geology, climatology, oceanography, etc.) 787 6.1
Civil, mechanical, chemical, or nuclear engineering 696 5.4
Medicine and/or Pharmacy 688 5.4
Social sciences 591 4.6
Chemistry 578 4.5
Economics/business 483 3.8
Psychology 418 3.3
Library and information science 375 2.9
High performance computing 365 2.8
Humanities 319 2.5
Education 264 2.1
Space sciences 163 1.3
Data Carpentry’s Respondents by Position n %
Graduate Student 592 45.9
Research Staff 200 15.5
Postdoctoral Researcher 183 14.2
Faculty 101 7.8
Government Employee 80 6.2
Industry Employee 49 3.8
Undergraduate Student 48 3.7
Management/Administrator 20 1.5
Retired/Not Employed 18 1.4

OS Respondents Use in Data Carpentry Workshops n %
Windows 661 53.3
Apple/Mac OS 512 41.3
UNIX/Linux 50 4.0
Not sure 17 1.4
Software Carpentry 1st Time Learners n %
Yes 11525 94
No 697 6
Data Carpentry’s U.S. Respondents’ Gender Identity n %
Female 322 58
Male 223 40
Transgender female 2 0
Prefer not to answer 8 1
Software Carpentry’s U.S. Respondents’ Gender Identity n %
Female 322 58
Male 223 40
Transgender female 2 0
Prefer not to answer 8 1
Data Carpentry’s U.S. Respondents Racial/Ethnic Identity n %
White 316 54.0
Asian 152 26.0
Hispanic or Latino(a) 57 9.7
I prefer not to say. 28 4.8
Black or African American 25 4.3
American Indian or Alaska Native 4 0.7
Native Hawaiian or Other Pacific Islander 3 0.5
Software Carpentry’s U.S. Respondents’ Racial/Ethnic Identity n %
American Indian or Alaskan Native 16 0
Asian / Pacific Islander 751 23
Black or African American 144 4
Hispanic or Latino 204 6
Native Hawaiian or Other Pacific Islander 3 0
White / Caucasian 1916 58
Prefer not to say 197 6
Multiple ethnicity / Other (please specify) 95 3

Summary